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Abstract 

In this paper, we give two explicit constructions of extractors, both of which work for a source of any 
min-entropy on strings of length n. The first extracts any constant fraction of the min-entropy using 
0(log 2 n) additional random bits. The second extracts all the min-entropy using 0(log 3 n) additional 
random bits. Both constructions use fewer truly random bits than any previous construction which works 
for all min-entropies and extracts a constant fraction of the min-entropy. 

The extractors are obtained by observing that a weaker notion of "combinatorial design" suffices 
for the Nisan-Wigderson pseudorandom generator [NW94], which underlies the recent extractor of Tre- 
visan [Tre98]. We give near-optimal constructions of such "weak designs" which achieve much better 
parameters than possible with the notion of designs used by Nisan-Wigderson and Trevisan. 

1 Introduction 

Roughly speakingran extractor is a function which extracts truly random bits from a weakly random 
sourcelusing a small number of additional random bits as a catalyst. A large body of work has focused on 
giving explicit constructions of extractorsras such constructions have a number of applications. A recent 
breakthrough was made by Luca Trevisan [Tre98]Twho discovered that the Nisan-Wigderson pseudorandom 
generator [NW94]Tpreviously only used in a computational settingrcould be used to construct extractors. 
Trevisan's extractor improves on most previous constructions and is optimal for certain settings of the 
parameters. Howeverrwhen one wants to extract all (or most) of the randomness from the weakly random 
sourcer Trevisan's extractor performs poorlyrin that a large number of truly random "catalyst" bits are 
needed. In this paperrwe give an extractor which extracts all of the randomness from the weakly random 
source using fewer truly random bits than any previous construction. This is accomplished by improving 
the combinatorial construction underlying the Nisan-Wigderson generator used in Trevisan's construction. 
Applying a construction of Wigderson and Zuckerman [WZ95]Twe also obtain improved expanders. 

Extractors. A distribution X on {0, l} n is said to have min-entropy k if for all x € {0, l}TPr [X = x] < 
2~ k . Think of this as saying that X has "k bits of randomness." A function Ext: {0, 1}™ x {0, l} d -» {0, l} m 
is called an (k, s)- extractor if for every distribution X on {0, 1}™ of min-entropy fcrthe induced distribution 
Ext(X, Ud) on {0, l} m has statistical difference at most e from uniform (where Ud is the uniform distribution 
on {0, l} d ). In other wordsrExT extracts m (almost) truly random bits from a source with k bits of hidden 
randomness using d additional random bits as a catalyst. The goal is to explicitly construct extractors which 
minimize d (ideallyrd = 0(log(n/e))) while m is as close to k as possible. 1 Dispersers are the analogue of 
extractors for one-sided error; instead of inducing the uniform distributionrthey simply hit all but a e fraction 
of points in {0, l} m . 



1 Actually, since the extractor is fed d truly random bits in addition to the k bits of hidden randomness, one can hope to 
have m be close to k + d. This will be discussed in more detail under the heading "Entropy loss." 



Previous work. Dispersers were first defined by Sipser [Sip88] and extractors were first denned by Nisan 
and Zuckerman [NZ96]. Much of the motivation for research on extractors comes from work done on "some- 
what random sources" [SV86rCG88rVaz87brVV85rVaz84rVaz87a]. There have been a number of pa- 
pers giving explicit constructions of dispersers and extractors! 1 with a steady improvement in the parame- 
ters [Zuc96nSfZ96rWZ95roW97rSZ98ESSZ98rNT98rTS98ITre98]. Most of the work on extractors is based 
on techniques such as fc-wise independencerthe Leftover hash lemma [ILL89]Tand various forms of composi- 
tion. A new approach to constructing extractors was recently initiated by Trevisan [Tre98]Twho discovered 
that the Nisan-Wigderson pseudorandom generator [NW94] could be used to construct extractors. 

Explicit constructions of extractors and dispersers have a wide variety of applicationsrincluding simu- 
lating randomized algorithms with weak random sources [Zuc96]; constructing oblivious samplers [Zuc97]; 
constructive leader election [Zuc97]; randomness efficient error-reduction in randomized algorithms and in- 
teractive proofs [Zuc97]; explicit constructions of expander graphsr superconcentratorsr and sorting net- 
works [WZ95]; hardness of approximation [Zuc96]; pseudorandom generators for space-bounded computa- 
tion [NZ96]; and other problems in complexity theory [Sip88rGZ97rACR97]. 

For a detailed survey of previous work on extractors and their applicationsrsee [NT98] . 

Our results. In this paperlVe construct two extractors: 

Theorem 1 For every n, k, m, ande, such that m <k <n, there are explicit (k,e)- extractors Ext: {0,1}™ x 
{0, l} d -» {0, l} m with 

L d - ° {lo g (k/m) ) 

2. d = (log 2 (n/e) log(l/7)), where 1 + 7 = k/(m - 1), and 1/m < 7 < 1/2. 

In particularising the first extractor with k/m constantrwe can extract any constant fraction of the 
source min-entropy using 0(log 2 n) additional random bitsrandr using the second extractor with k = mT 
we can extract all of the source min-entropy using 0((log 2 n)(log k)) additional random bits. A comparison 
of these extractors with the best previous constructions is given in Figure 1. Our second extractor directly 
improves that of Ta-Shma [NT98]Tin that ours uses 0((log 2 n)(log k)) < 0(log 3 n) truly random bits in 
comparison to a polynomial of unspecified (and presumably large) degree in logn. Both of our extractors 
use more truly random bits than the extractors of [Zuc97rTre98] and the disperser of [TS98]Tbut our 
extractors have the advantage that they work for any min-entropy (unlike [Zuc97]) and extract all (or a 
constant fraction) of the min-entropy (unlike [TS98rTre98]). The disadvantage of the extractors of [GW97] 
described in Figure 1 is that they only use a small number of truly random bits when the source min-entropy 
k is very close to the input length n (e.g.Tk = n — polylog(n))r whereas ours uses 0(log 3 n) random bits 
for any min-entropy. There are also extractors given in [GW97FSZ98] which extract all of the min-entropyr 
but these use a small number of truly random bits only when the source min-entropy is very small (e.g.T 
k = polylog(n))rand these extractors are better discussed later in the context of entropy loss. 

Plugging our second extractor into a construction of [WZ95] immediately yields the following expander 
graphs: 

Corollary 2 For every N and K < N , there is an explicitly constructible graph on N nodes with degree 
(N/K) ■ 2°(( loglog N > ( l °(L lo (L K )) such that every two disjoint sets of vertices of size at least K have an edge 
between them. 

This degree compares with a degree bound of (N/K) • 2°(P ol y(iogiog N )) due to Ta-Shma [NT98]. Such 
expanders have applications to sorting and selecting in roundsrconstructing depth 2 superconcentratorsrand 
constructing non-blocking networks [Pip87rAKSS89r\VZ95]. 

The Trevisan extractor. The main tool in the Trevisan extractor is the Nisan-Wigderson genera- 
tor [NW94]T which builds a pseudorandom generator out of any predicate P such that the security of the 
pseudorandom generator is closely related to how hard P is to compute (on average). Let S = (Si, . . . , S m ) 
be a collection of subsets of [d] each of size ffand let P: {0, 1}* — > {0, 1} be any predicate. For a string 
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(Abovera is an arbitrarily small constant.) 
Figure 1: Comparison with best previous constructions 



y € {0, l} d rdefine y\s { to be the string in {0, 1}* obtained by projecting y onto the coordinates specified by 
Si. Then the Nisan-Wigderson generator NW^p: {0, l} d — > {0, l} m is defined as 

NW 5 ,p(i/)=P(i/| Sl )---P(i/| Sm ). 

In the "indistinguishability proof" of [NW94]Tit is shown that for any function D:{0,l} m -» {0,1} 
which distinguishes the output of NW^^fj/) (for uniformly selected y) from the uniform distribution on 
{0,l} m rthere is a "small" circuit C (or procedure of small "description size") such that C D '(•) (i.e. TC 
with oracle access to D) approximates P(-) reasonably well. It is shown that the size of the C is related 
to max,^j \Si n Sj|rso one should use a collection of sets in which this quantity is smalir while trying to 
minimize the seed length d. 

We now give a rough description of the Trevisan extractor Ext: {0, l} n x {0, l} d — > {0, l} m . For a string 
u € {0, l}Tlet u £ {0, 1}™ be an encoding of u in an error-correcting code (whose properties are unimportant 
in this informal description) and define t = logn. We view u as a boolean function u: {0, 1}* — > {0, 1}. 

Then the extractor is simply 

ExT S (u,y) =NW < s i n(2/) =u(y\ Sl )---u(v\s m )- 

The analysis of this extractor in [Tre98] shows that the output of this extractor is close to uniform as 
long as the source min-entropy required is greater than the size of the circuit built in the security reduction 
of [NW94] . Hencerone needs to keep this circuit size small while minimizing the number d of truly random 
bits neededr which is equal to the seed length of the Nisan-Wigderson generator. UnfortunatelyT using 
maxj^j \Si n Sj\ as the measure of the circuit size as in [NW94rTre98]rone cannot make d much smaller 
than what is obtained in [Tre98]. 

The improvement. The improvements of this paper stem from the observation that actually max, ^ j<« 2^ SinS ' I 
is much better than max^j \Si fl Sj| as a measure of the size of the circuit built in the Nisan-Wigderson se- 
curity reduction. So we are left with the problem of constructing set systems in which this quantity is small; 
we call such set systems weak designs (in contrast to designsTm which max^j |5,n5j| is bounded). We show 
that with weak designsrone can have d much smaller than is possible with the corresponding designs. The 
weak designs used in our first extractor are constructed using an application of the Probabilistic Methodr 
which we then derandomize using the Method of Conditional Expectations (see [ASE92] and [MR95rCh. 
5]). We then apply a simple iteration to these first weak designs to obtain the weak designs used in our 
second extractor. We also prove a lower bound showing that our weak designs are near-optimal. 

Entropy loss. Since a (A;,e)-extractor Ext: {0, 1}™x{0, l} d — > {0, l} m is given A; bits of hidden randomness 
in its first input and d truly random bits in its second inputrone can actually hope for the output length 



m to be almost k + drrather than just k. The quantity A = k + d — to is therefore called the entropy loss 
of the extractor. Hencerin this languagerthe goal in constructing extractors is to simulataneously minimize 
both d and the entropy loss. 

NonconstructivelyTone can show thatrfor any n and k < nT there exist extractors Ext„^: {0, 1}™ x 
{0, l} d -» {0, l} k + d ~ A with d = log(n -k) + 0(1) and entropy loss A = 2 log(l/e) + 0(l)rand these bounds 
on d and A are tight up to additive constants [RT97]. The explicit constructionsrhoweverrare still far from 
achieving these parameters. As for what is knownrevery entry in Figure 1 with m = k has an entropy loss of 
d. For examplerthe extractor of [GW97] has an entropy loss of 0(n — k + log(l/e)) (which is only interesting 
when k is very close to n) and the extractor of [NT98] has an entropy loss of polylogn. In additionrthe 
"tiny families of hash functions" of [GW97rSZ98] give extractors with d = 0(k + logn) and entropy loss 
21og(l/e) + 0(1); these are interesting when k is very small (e.g.Tk = polylogn) 

A slight modification to our second extractor enables us to achieve logarithmic entropy loss: 

Theorem 3 For every n, k, and s such that k < n, there is a (k,e)- extractor Ext: {0, 1}™ x {0, l} d — > 
{0, l}*+ d " A with 

d = ((log 2 (n/e))(logfc)) 

and entropy loss 

A = 31og(fc/e) + 0(l) 

2 Preliminaries 

In this sectionrwe introduce some standard terminology and notation used throughout the paper, log 
indicates the logarithm base 2 and In denotes the natural logarithm. If X is a probability distribution on a 
finite setrwe write x±- X to indicate that x is selected according to X. Two distributions X and Y on a set 
S are said to have statistical difference (or variation distance) e if 

max |Pr [D{X) = 1] - Pr [D(Y) = 1]| = e, 

where the maximum is taken over all functions ("distinguishers") D: {0, l} m — > {0, 1}. A distribution X is 
said to have min-entropy k if for all xTPt [X = x] < 2~ h . It is useful to think of distributions of min-entropy 
k as being uniform over a subset of the domain of size 2 k . 

We write Uj for the uniform distribution on strings of length j. A function Ext: {0, 1}™ x {0, l} d — > 
{0, l} m is a (k, e)-extractor if for every distribution X of min-entropy ATExt(X, Ud) has statistical difference 
at most e from U m . In other wordsrExT extracts m (almost) truly random bits from a source with k bits 
of hidden randomness using d additional random bits as a catalyst. We say that a family of extractors 
{Ext,: {0, l} ni x {0, 1} i — > {0, l} mi }, G / is explicit if Ext, can be evaluated in time poly(rij,dj). 

3 Combinatorial designs 

The combinatorial construction underlying the Nisan-Wigderson generator are combinatorial designs. 
Definition 4 ([NW94]) 2 A family of sets Si, . . . , S m C [d] is an (£,p)-design if 

1. For alii, |S;| = £. 

2. For all i ^ j, \S t D Sj\ < log p. 



2 There is a somewhat related notion in the combinatorics literature known as a 2-design (see, e.g. [AK92]). In 2-designs, 
strong additional regularity requirements are imposed (such as all the pairwise intersections being exactly the same size and all 
points being contained in the same number of sets). These additional requirements are irrelevant in our applications. 



Motivation. In Trevisan's extractoiT the parameters of a design correspond to the parameters of the 
extractor as follows: 



source min-entropy ps pm 

output length = m 

input length = 2 ew 

additional randomness = d 

Hencelbur goal in constructing designs is to minimize d given parameters TOHTand p (such that p > 1). 
Notice that 1/p is essentially the fraction of the source min-entropy that is extractedrso ideally p would be 
as close to 1 as possible. 

One explicit construction of designs is given by the following: 

Lemma 5 ([NW94, Tre98]) For every m, £, and p > 1, there exists an efficiently constructible (£,p)- 
design Si, ... , S m C [d] with 

£2 m O(l/lo g/ 9) 

d= . 

logp 

Notice that the dependence on p is very poor. In particularly we want to extract a constant fraction 
of the min-entropyTwe need more than m c truly random bits for some c > 0. This is unavoidable with the 
current definition of designs: if p < 2rthen all the sets must be disjointrso d > m£. In generallwe have the 
following lower boundrobtained in joint work with Luca Trevisan and proved in Section 6: 

Proposition 6 If Si,.. . ,S m C [d] is an (I, p) -design, then 

d> m l/log2 P .(£_logp) 

The improvements of this paper stem from the observation that actually a weaker form of design suffices 
for the Nisan-Wigderson generator and the construction of extractors: 

Definition 7 A family of sets Si , . . . , S m C [d] is a weak (£, p)-design if 

1. For alii, \S t \ = £. 

2. For all i, 

Y,2 lSinSil <p-(m-l). 

j<i 

We will show that the parameters of a weak design correspond to the parameters of our extractors in 
the same way that designs corresponded to the parameters of Trevisan's extractor. Notice that every (£,p)- 
design is a weak (£, p)-design. But one canlTor many settings of mrffand p achieve weak (£, p)-designs 
Si, . . . ,S m C [d] with much smaller values of d than possible with (£, p)-designs. IndeedlVe will prove the 
following in Section 5 using a probabilistic argument: 

Lemma 8 For every £, m € N and p > 1, there exists a weak (£,p)-design Si, . . . ,S m C [d] with 



lnp 

Moreover, such a family can be found in time po\y(m,d). 

This is already much better than what is given by Lemma 5; for constant pTd is 0(£ 2 ) instead of £ 2 m c . 
Howeverras p gets very close to lTd gets very large. Specificallyrif p = 1 + 7 for small 7rthen the above 
gives d = 0(£ 2 l^i). To improve thisrwe notice that the proof of Lemma 8 does not take advantage of the 
fact that there are fewer terms in '^2j <i ^ SinS ^ when i is small; indeed the proof actually shows how to 



obtain Yjj<h 2^ SinS ^ < p ■ {i — 1) with the same d. 3 Since we only need a bound of p ■ (to — 1) for all iT 
this suggests that we should "pack" more sets in the beginning. This packing is accomplished by iterating 
the construction of Lemma 8 (directly inspired by the iteration of Wigderson and Zuckerman [WZ95] on 
extractors) Tan d yields the following improvement. 

Lemma 9 For every I, to € N and 3/to < 7 < 1/2, there exists a weak (to, 1,1 + 7, d)-design with 

d = OU 2 log - 

Moreover, such a family can be found in time poly(m,<f). 

In particularLwe can take 7 = 6(1/to) and extract essentially all of the entropy of the source using 
d = 0(l 2 log to) truly random bits. Lemma 9 will be proven in Section 5. 

For extractors which use only O(logn) truly random bitsF where n is the input lengthFone would need 
d = 0(1). HoweverFone cannot hope to do better than fl(l 2 ) using the current analysis with weak designs. 
IndeedFthe following proposition shows that our weak designs are optimal up to the log(l/7) factor in our 
second construction. 

Proposition 10 For every (l,p)-weak design Si, . . . ,S m C [d], 

I 2 ml 



d > min 



21og2p' 2 



Notice that d = ml can be trivially achieved having all the sets disjoint and that log 2p approaches 1 as 
p approaches IXso the lower bound for p « 1 is essentially fl(l 2 ). 

4 The extractor 

In this sectionrwe describe the Trevisan extractor and analyze its performance when used with our weak 
designs. The description of the extractor follows [Tre98] very closely. The main tool in the Trevisan extractor 
is the Nisan- Wigderson generator [NW94]. Let <S = (Si, . . . , S m ) be a collection of subsets of [d] of size IT 
and let P: {0, 1}* — > {0, 1} be any boolean function. For a string y € {0, l} d rdefine y\s t to be the string in 
{0, 1}* obtained by projecting y onto the coordinates specified by S,. Then the Nisan- Wigderson generator 
NW^p is defined as 

NW 5 ,p(i/)=P(i/| Sl )---P(i/| Sm ). 

In addition to the Nisan- Wigderson generator rthe Trevisan extractor makes use of error-correcting codes: 

Lemma 11 (Error-correcting codes) For every n and S there is a code EC„^:{0,1}™ — > {0,1}™ where 
n = poly(n, 1/S) such that every Hamming ball of relative radius 1/2 — S in {0, 1}™ contains at most 1/S 2 
codewords. Furthermore, EC„^ can be evaluated in time poly(n,l/<5) andn can be assumed to be a power 
of 2. 

We will actually use a stronger property of such error-correcting codes: 

Proposition 12 Let EC„^ be any family of codes such that every Hamming ball of relative radius 1/2 — S 
contains fewer than B codewords and h\B < S 2 • n. Then, for sufficiently small S, every l\-ball of relative 
radius 1/2 — 28 in [0, 1]™ C IK™ contains at most B codewords. In other words, for every real vector v € [0, 1]™, 



*\ u =i2\ Ec ( u ^- v i\<l- 2s \ <B 



3 In fact it is necessary that d = Q.(t 2 /\ogp) if 5Z,-<i ^ SinSj < p ■ (i — 1) for all i. See the remark after the proof of 
Proposition 10. 



Since we may assume that n > S 2 ln<5 2 in Lemma llX Proposition 12 applies to those codes with 
B = 1/S 2 . The proof of Proposition 12 was obtained with the help of Madhu Sudan. 

Proof: Suppose notr i.e. there are at least B codewords in an £i-ball of radius 1/2 — 26 around some 
real vector v. Thenr consider a vector w € {0,1}™ obtained by randomly rounding v. That isriet w, = 1 
with probability Vi for every i independently. Now consider any codeword u within t\ -distance 1/2 — 25 of 
v. The Hamming distance between u and w has expectation ||u — v\\\ and is the sum of n i.i.d. [0, l]-valued 
random variables. By the Hoeffding inequalityrthe probability that the Hamming distance between u and 
w exceeds ||IT — i»||i +S is at most exp(— 2n<5 2 )) = 1/|5| 2 . Hencerthe probability that there are fewer than B 
codewords within Hamming distance 1/2 — S around w is at most 1/|B|. Thusrthere exists a w with at least 
B codewords within Hamming distance 1/2 — S contradicting the property of the error-correcting code. ■ 

We can now describe the Trevisan extractor Pwhich takes as parameters nFmrfcrand erwhere m <n < k. 
Let EC: {0,1}™ -> {0,1}™ be as in Lemma 12Lwith 6 = e/4m and define I = logn = 0(logn/e). For 
u € {0, l}"Fwe view EC(u) as a boolean function u: {0, 1}* — > {0, 1}. 

Let S = (S\, . . . ,S m ) be a collection of subsets of [d] (for some d) such that |S,| = t for each i. (How 
S is selected will crucially affect the performance of the extractor; we will later choose it to be one of our 
weak designs.) 

Then the extractor Ext,s: {0, 1}™ x {0, l} d ->• {0, l} m is defined as 

ExT S (u,y) = NW < s i n(2/) =u(y\ Sl )---u(v\s m )- 

We will now analyze this extractor. The following Lemma is implicit in [NW94] and is more explicitly 
shown in [Tre98]. It shows howFfrom any function D which distinguishes the output of NW^p from uniformF 
one can obtain a (randomized) "program" whichFusing D as an oracleFpredicts P with noticeable advantage. 
This lemma shows that this "program" can be taken to be of a very simple formF which will allow us to 
bound its complexity later on. The only modification we need to the proofs of [NW94FTre98] is that we do 
not use an averaging argument to fix the "random part" of the hybrids in the "hybrid argument" ; ratherF 
we keep these random. 

Lemma 13 Let P: {0, 1}* — > {0, 1} be any predicate and let D: {0, l} m — > {0, 1} be any "distinguisher" such 
that 

Pr [D(r) = 1] - Pr [D(NW 5 ,p(l/)) = 1] > e, 

r y 

where r is selected uniformly from {0, l} m and y from {0, l} d . Then there exists a there exists an i <m and 
functions Pi , . . . , Pi from {0, 1}* to {0, 1} 

1. Pr, x i, r [D(Pi(x), . . . ,Pi-i(x),b,r) (Bb = P(x)] > \ + ^, where x is selected uniformly from {0,1}^, b 
from {0, 1}, and r from {0, l} m- \ 

2. Each Pj depends on only \Si PI Sj\ bits of x (where these bit positions depend only on S and i, but not 
on P or D) 

Proof sketch: We can expand the hypothesis of Lemma 13 as 

Pr [D (n ■ ■ ■ r m ) = 1] - Pr [D (P(y\ Sl ) ■ ■ ■ P(y\sJ) = 1] > e, 

where n, . . . , r m are uniformly and independently selected bits and y is uniformly selected from {0, l} d . By 
the "hybrid argument" of [GM84] (cf. [Gol95rSec. 3.2.3])rthere is an i such that 

Pr [D(P(y\ Sl )---P(y\ Si _ 1 )r i ---r m )=l\- Pr [D (P(y\ Sl ) ■ ■ ■ P(y\ Si )r i+ i ■ ■ ■ r m ) = 1} > e/m 

NowFrenaming r, as b and using the standard transformation from distinguishers to predictors [Yao82] (cf. 
[Gol98rSec. 3.3.3])rwe see that 

Pr [D(P(y\ Sl )---P(y\ Si _ 1 )br i+ i---r m )(Bb = P(y\ Si )]>\ + - 

y,b,n + i---r m Z m 



Using an averaging argument we can fix all the bits of y outside 5, while preserving the prediction advantage. 
Renaming ys t as xTwe now observe that x varies uniformly over {0, 1}* while P(y\sj) for j ^ i is now a 
function Pj of x that depends on only IS", (~l Sj\ bits of x. Sorwe have 

Pr [D (Pi (a;) ■ ■ ■ p_ x (a:)&r i+ i • • • r m ) © fo = P{x)] > \ + — , 

x,b,ri + i---r m Z Til 

as desired. □ 



Now we use a counting argument to bound the complexity (or "description size" ) of the "program" above 
and illustrate the connection with weak designs: 

Lemma 14 There is a set T of functions from {0, l|^+ 1 + m t {0, l} m (depending only on S) such that 

1. For every predicate P: {0, 1}* — > {0, 1} and distinguisher D: {0, l} m — > {0, 1} such that 

Pr [D(r) = 1] - Pr [D(NW 5 P (y)) = 1] > e, 

r y 

there exists a function f € T such that 

Pr [D(/(a;, 6, r)) ©6 = P(a;)]>^ + -, 

where x is selected uniformly from {0, 1}^, b from {0, 1}, and r from {0, l} m . 

2. log |JF| < logm + maxj ^j<i 2\ SinS ^ . 

3. Each function in T can be computed by a circuit of size O (max, (^ 7< j \Si (~l Sj\ • 2> SinSj > J J . 4 

We will not use Item 3 (the bound on circuit size) in the analysis of our extractor; we only use this for 
our quantitative improvement to the pseudorandom generators of [NW94] given in Section 8. 

Proof: By Lemma 13rto meet Condition 1 it suffices to let T be the set of functions / of the form 
(x, b, r) h-> (Pi (x) , Pi (x) , . . . , Pj_i (x) , b, r)Twhere Pj (x) depends only some set T t j of bits of a;Pwhere \T t j \ = 
\Si n Sj|. The number of bits it takes to represent i is logm. Given iT the number of bits it takes to 
represent each Pj is 2l Ti3 'l = 2\ SinS '\. Sorthe total number of bits it takes to represent a function in T is 
logm + max, ^2j<i 2l' Sin ' S: 'lrgiving the desired bound on log \T\. 

For the bound on circuit sizeLnotice that the circuit size of / is simply the sum of the circuit sizes of the 
Pj'srand every function on k bits can be computed by a circuit of size 0(k2 k ). ■ 

We now analyze the extractor Ext^ when we take S to be a weak design. The argument follows the 
analysis of Trevisan's extractor in [Tre98] except that we use the more refined bounds on \F\ given by 
Lemma 14. 

Proposition 15 If S = (Si, . . . ,S m ) (with 5, C [d]) is a weak (£,p)-design for p = (k — 31og(m/e) — 5)/m 
(where c is a fixed constant), then Ext,?: {0, 1}™ x {0, l} d — > {0, l} m is a (k,e)- extractor. 

Proof: Let X be any distribution of min-entropy k. We need to show that the statistical difference between 
U m and Ext(X, Ud) is at most e. By the definition of statistical differencerthis is equivalent to showing 
thatrfor every distinguisher D: {0, l} m ->• {0, l}r 

Pr [D(r) = 1] - Pr [D(NW s ,u(y)) = 1] < e 



We measure circuit size by the number of internal gates, so, for example, the identity function has circuit size 0. 



where r and y are selected uniformly from {0, l} m and {0, l} d rrespectively. (We have dropped the absolute 
value in the definition of statistical difference; this is without loss of generality since we may replace D by 
its binary complement.) So let D: {0, l} m — > {0, 1} be any distinguisher and let T be as in Lemma 14rso 
\T\ < m2 pm . For every / € JTwe obtain a function /: {0, 1}* — > [0, 1] given by 

f(x)=Pr[D(f(x,b,r))®b = l]. 

b,r 

Think of D(f(x, b, r)) ffi b as a randomized algorithm built out of D with input x and random coins (b, r). 
We can view each / as a vector / e [0, 1]™. Notice that for a predicate P: {0, 1}* — > {0, 1} (which we can also 
view as a vector P e {0, l}")rthe ^-distance between / and P gives the probability that the randomized 
algorithm corresponding to / computes P incorrectly. That isr 

\f~P\i = P*[D(f(x,b,r)) © b ? P(x)}. 

b,r 

Let B be the set of u for which there exists an / € T such that \f — u\\ < 1/2 — s/2m. In other wordsr 
B is the set of "bad" u for which u can be approximated easily by one of these randomized algorithms /. 
By the property of the error-correcting code given in Proposition 12Ffor each function / € .FFthere are at 
most (4m/e) 2 strings u € {0, 1}™ such that \f — u\\ < 1/2 — s/2m. By the union boundF 

\B\ < (4m/e) 2 • |.F| = (4m/e) 2 • 2 pm . 

Since X has min-entropy fcFeach u € B has probability at most 2~ k of being selected from XTso 

\'2^npm\ i^ — k 



Pr [u £ B] < ((4m/e) 2 m2 pm ) ■ 2" 



= ((4m/£) 2 m2*- 31 °s(™/ £ »- 5 ) • 2" 
< e/2 



■k 



NowFby Lemma 14Fif u £ BFthen 



Pr [D(r) = 1] - Pr[ J D(NW <s ,^( ?/ )) = 1] < e/2. 

r y 



ThusF 



Pr[D(r)=l]- Pr [fl(NW s , s (y)) = 1] = E 

r u-i-X,y ' " u<-X 



Pr[D(r) = 1] - Pr [D(NW 5 , 5 (j/)) = 1] 

r 

i£B] + 
< e/2 + e/2 = e. 



< Pr [u G B] + Pr [u ^ B] ■ e/2 



Combining Proposition 15 with the weak designs given by Lemmas 8 and 9 essentially proves Theorem 1. 
The only technicality is that Proposition 15 does not allow us to take p = k/m (or k/(m — 1)) which is what 
we would need to deduce Theorem 1 directly. InsteadFwe lose A = 3 log(m/e) + 5 bits of the source entropy 
in Proposition 15. HoweverFsince A is so smallFwe can give our extractor A more truly random bits in its 
seed (increasing d by only a constant factor) which we just concatenate to the output to compensate for the 
loss. The details of this are given below. 

Proof of Theorem 1: Let A = 31og(m/e)+5. Let k' =k-ATm' = m-A-3Fand p = k 1 /to' > fc/(m-l). 
For 1 or 2rapply Proposition 15 with the weak (£, p)-design Si, . . . , S m < C [d 1 ] of Lemma 8 or Lemma 9r 

respectively. This gives an (A;,e)-extractor Ext: {0, 1}™ x {0,l} d ' -> {0,l} m 'rwith d' = O ( ^fpyffi ) or 

d' = 0(log 2 (n/e)log(l/7))rrespectively By using A + 3 additional bits in the seed and simply concate- 
nating these to the outputrwe obtain a (A;,e)-extractor Ext: {0, 1}™ x {0, l} d + A + 3 -> {0, l} m ras desired. 



(In applying Lemma 9rwe need to make sure that p < 3/2rbut if p > 3/2Twe can use the weak design of 
Lemma 8 instead.) ■ 



Remark The stronger property of error-correcting codes given by Proposition 12 which corresponds to 
hardness against randomized algorithms could have been avoided by using an averaging argument to "fix" r 
and b in Lemma 13. If this is done in a straightforward mannerLwe would have to pay a price for these bits 
in the size of JFLas they would be needed to fully describe a function. The cost of these bits can be avoidedL 
howeverLif we do the hybrid argument while we are still looking at the advantage of the distinguisher averaged 
over the choice of u; then these bits can be fixed independently of u and absorbed into the distinguisher 
before we make the counting argument which says that the distinguisher fails with probability at least 1 — e/2 
over the choice of u-^- X. Doing the analysis this way eliminates the logm term in the bound on log|.F|. 
HoweverLthe approach we have takenL advocated by Oded GoldreichL corresponds better to the intuition 
that one should not pay a price for r and b since they can be taken to be random. 



5 Construction of weak designs 



. We view [d] as the disjoint union 
. , S m in sequence so that 



Proof of Lemma 8: Let ffmLand p be givenLand let d = \tf lnp] ■ 
of £ blocks Si, ... , Bi reach of size \tf lnp] . We construct the sets Si, . 

1. Each set contains exactly one element from each blockrand 

2- Ei«2l s ' ns 'l<p-(t-l). 

Supppose we have Si,...,S,_i C [d] satisfying the above conditions. We prove that there exists a 
set Si satisfying the required conditions using the Probabilistic Method [ASE92] (see also [MR95r Ch. 
5]). Let ai, . . . , ai be uniformly and independently selected elements of B\, . . . , B^rrespectivelyPand let 
Si = {ai, • • • , ai}. We will argue that with nonzero probabilityTCondition 2 holds. Let Y,-^ be the indicator 
random variable for whether au £ S,Tso Pr [Y,-^ = 1] = l/\Bj\ = l/|~£/lnp~|. Notice that for a fixed jTthe 
random variables Y^i, . . . , Y^t are independent. 



E 



E 2lSl 



nSj\ 



j<i 



5>[2^ 



j<i 



Ee 



IF 



j<i L k 

= EIIe^-] 

j<i k 

< (i-l)-P 



\l/lnp] 



Hencerwith nonzero probabilityrCondition 2 holdsrso a set S, satisfying the requirements exists. How- 
everPwe want to find such a set deterministically. This can be accomplished by a straightforward application 
of the Method of Conditional Expectations (see [ASE92] and [MR95rCh. 5]). Details can be found in 
Appendix A. ■ 



Remark A perhaps more natural way to carry out the above probabilistic construction is to chose S, 
uniformly from the set of all subsets of [d] of size ffrather than dividing [d] into t blocks. This gives essentially 
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the same boundsrbut complicates the analysis because the elements of 5, are no longer independent. The 
cleaner approach in the above proof was suggested by David Zuckerman. 

Proof of Lemma 9: For simplicityTassume that I+7 = 1/(1 — 2~ h ) and m = 2 9 /(l+^f). Let do = [£/ln2]-£ 
and let d = h- do = 0(£ 2 ■ log(l + 7)). We view [d] as the disjoint union of h blocks B\, . . . , Bh each of size 



TO. 



do- For each t € [/i]riet m t = 2 9 ~ t and n t = 5] s =i w^Tso J2 t m t 

Now we define our weak design Si, . . . ,S m . For each t € [/ijTwe let S nt +i, . . . ,S nt + mt C B t be a weak 
(£, 2)-design as given by Lemma 8. In other wordsrwe take the ordered union of h weak (£, 2)-designs 
(consisting of toi,to,2, . . . ,m/, setsFrespectively) using disjoint subsets of the universe for each. The number 
of sets is TO,rthe size of the universe is drand each set is of size ffso we only need to check that for all i € [to]T 
Sj<j 2\ SinS '\ < p ■ (to — 1). For i £ {n t + 1, . . . ,n t + m t }TSi is disjoint from any Sj for any j < n t and 

i-\ 

J2 2\ s ' ns '\ <2-(m t -l). 

j—nt + l 

since S„,+i, . . . , S nt+mt is a weak (£, 2)-design. 
Thusrwe have 

n t i — l 

Si\ 



j<i j—1 j=n t + l 

< n t +2-(m t -l) 

= 2«-2<(l+7)(m-l), 



as desired. 



6 Lower bounds for designs 

Proof of Proposition 6: Let I = max,^ |S, (~l Sj\ < logp. For each j = 1, . . . , TOFlet Tj be the set of 

subsets of Sj of size I + ITso \Tj\ = ( I+1 )- Let T = M. Tj. Notice that the sets Tj are disjointrbecause no 
two distinct sets Sj,Sj share more than I elements. Thusr|r| = to • (jij- At the same timer|r| consists of 
subsets of [d] of size I + ITso \T\ < ( I+1 ) ■ So we have 

I+lJ -\I + 1 
Expanding the binomial coefficients and rearranging termsrwe have 



I) \l-IJ - \£-IJ -\£-logp 



Proof of Proposition 10: We have 

p > max V 2 |Sin ^ I 

i TO- 1 ^ 



TO 



- m 

~ to(to -\) *-*'*-*' 
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2to(to - 1) -^ 

where the last inequality follows from Jensen's inequality. Thusr 

log2p>-LY,\ S i nS i\ W 






Nowrfor a G [djriet n a = \{i: a G Si}\. Then Y. a n a =J2i\Si\ = m ■ 

^l^n^-l = £> a ( na -i) 

a6[d] 

a a 



^ 2 yl na ) ~ ml 



> 



ra 2 t 2 

— ; rat 

a 

ra 2 £ 2 



2d ' 
unless d > (mtj/2. Putting this in Inequality lrwe have 

, „ 1 m 2 £ 2 £ 2 

l0g2p> ^-^d- = 2d 

which proves the proposition. ■ 



Remark The above proof gives a stronger bound on d if we have a family of sets Si , . . . , S m such that for 
all iT ^2 j Ki 2\ SinSj \ < p ■ {i — 1) (e.g.Tthe family of sets constructed in the proof of Lemma 8). If we have 
such a boundrthen summing over i from 1 to to gives 



,.(")> I J>«n„, 



(I 

and applying Jensen's inequality and taking logs as in the above proof gives 



lo s^>iEi^ n5 ii 



TO/ 



instead of Inequality 1. Following the rest of the proof without changerthis shows that 

I 2 rat 



d > min 



21ogp 2 
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7 Achieving small entropy loss 

Recall that the entropy loss of an extractor Ext: {0, 1}™ x {0, l} d -> {0, l} m is defined as A = k + d - mV 
and we can hope for this to be as small as 21og(l/e) + 0(1) with d = log(n — k) + 0(1) [RT97]. 

In constructing our extractor Ext,s(w, y) = NW^^Fwe "threw away" y after using it as a seed for the 
Nisan-Wigderson generator and hence the d bits of entropy carried by y were lost. However rthe analysis of 
the Nisan-Wigderson generator actually shows that the quality of the generator is not affected if the seed 
is revealed. Thusrwe define ExT' s (u,y) = fcNW^fj/)). Now all the analysis of Ext done in Section 4 
actually applies to Ext' (in Lemmas 13 and 14rgive the distinguisher D the seed y in addition to NW5 i5 (j/))r 
and we obtain the following strengthening of Proposition 15: 

Proposition 16 If S = (Si , . . . , S m ) (with S, C [d]) is a weak (£, p)-design for p = (k — 3 log(m/e) — 5)/m, 
then Ext^: {0, 1}™ x {0, l} d -» {0, \} m + d is a (k, e)-extractor. 

Combining Proposition 16 and Lemma 9 with m = k — 1 immediately gives Theorem 3. An additional 
additive factor of log m can be removed from the entropy loss by taking the alternative approach mentioned 
in the remark at the end of Section 4. Note that the trick of adding extra bits to the seed and concatenating 
these to the outputLas we did in the proof of Theorem lLdoes not help in reducing the entropy loss. 

8 Better pseudorandom generators 

Using alternative types of designs also gives some quantitative improvements in the construction of pseu- 
dorandom generators from hard predicates in [NW94] . From Lemma 14Fwe see that the relevant notion of 
design in the setting of circuit complexity-based pseudorandom generation is the following: 

Definition 17 A family of sets Si, ... , S m C [d] is a type 2 weak (£,p)-design if 

1. For alii, \S t \ = £. 

2. For all i, 

^2\SinSj\-2\ s ' nS '\ <p-(m-l). 

j<i 

Notice that it is meaningful to consider even values of p less than lFsince |S, PI Sj\ ■ 2\ SinS: >\ can be zero. 
Using a construction like the one in Lemma 8Fwe obtain 

Lemma 18 For every £, m £ N and p> 0, there exists a type 2 weak (£,p)-design Si,.. . ,S m C [d] with 



0(j) *fP< 



d 



6 



Moreover, such a family can be found in time po\y(m,d). 

The quantitative relation between pseudorandom generators and type 2 weak designs follows readily from 
Lemma 14: 

Lemma 19 Suppose P: {0, 1}* — > {0, 1} is a predicate such that no circuit of size s can compute P correctly 
on more than a fraction \ + e of the inputs and suppose that S = (Si , . . . , S m ) where S, C [d] is a type 
2 weak (£,p)-design. Then no circuit of size s — pm can distinguish NW^p from uniform with advantage 
greater than me. 

Combining this and Lemma 18 with p = 1 and s = 2mFwe obtain 
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Theorem 20 Suppose P: {0, 1}* — > {0, 1} is a predicate such that that no circuit of size 2m can compute P 
correctly on more than a fraction \ + ^ of the inputs. Then there is a generator Gp :m : {0, 1}°" / lo s^) — > 
{0, l} m computable in time po\y(m,£), making m oracle calls to P, such that no circuit of size m can 
distinguish the output of G from uniform with advantage greater than e. 

In other wordsrto obtain m bits which are pseudorandom against circuits of size mTwe need only assume 
that there is a predicate which is hard against circuits of size 0(m). In contrastrthe results of [NW94] 
always need to assume that the predicate is hard against circuits of size m 1+e for some constant e > (or 
else their generator will require a seed length that is polynomial in m instead of £). In factrif we instead 
take p = 1/ffwe need only assume that the predicate is hard against circuits of size (1 + l/£) ■ m (and the 
generator will have a seed length 0(£ 2 )). 

Acknowledgments 

I am grateful to Luca Trevisan for sharing his novel insights into these problems with me. I acknowledge 
Oded GoldreichrMadhu Sudanrand David Zuckerman for several simplifications of the proofs in this paper 
and for helpful discussions. Further thanks to Oded Goldreich for valuable comments on the presentation. 

References 

[ACR97] Alexander E. AndreevI 1 Andrea E. F. Clementirand Jose D. P. Rolim. Worst-case hardness 
suffices for derandomization: A new method for hardness-randomness trade-offs. In Pierpaolo 
DeganorRobert Gorrierirand Alberto Marchetti-Spaccamelareditorsr^Miomaia, Languages and 
Programming, 24th International ColloquiumTvolume 1256 of Lecture Notes in Computer ScienceT 
pages 177-187rBolognantalyr7-ll July 1997. Springer- Verlag. 

[AK92] E.F. Assmus and J.D. Key. Designs and their codes. Number 103 in Cambridge Tracts in Math- 
ematics. Cambridge University Pressri992. 

[AKSS89] Miklos Ajtairjanos KomlosrWilliam Steigerrand Endre Szemeredi. Almost sorting in one round. 
In Silvio Micalir editorr Randomness and ComputationT volume 5 of Advances in Computing 
ResearchTpages 117-125. JAI Press Inc.ri989. 

[ASE92] Noga Alonrjoel H. Spencerrand Paul Erdos. The Probabilistic Method. Wiley-Interscience Series 
in Discrete Mathematics and Optimization. John Wiley and Sonsrinc.ri992. 

[CG88] Benny Chor and Oded Goldreich. Unbiased bits from sources of weak randomness and probabilistic 
communication complexity. SIAM Journal on Co77y)M£m<7ri7(2):230-26irApril 1988. 

[GM84] Shan Goldwasser and Silvio Micali. Probabilistic encryption. Journal of Computer and System 
S«encesr28(2):270-299rApril 1984. 

[Gol95] Oded Goldreich. Foundations of Cryptography (Fragments of a Book). Weizmann Institute of Sci- 
enceri995. Availableralong with revised version l/98Pfrom http://theory.lcs.mit.edu/~oded. 

[Gol98] Oded Goldreich. Modern Cryptography, Probabilistic Proofs and PseudorandomnessT June 1998. 
Available from http://theory.lcs.mit.edu/~oded/. 

[GW97] Oded Goldreich and Avi Wigderson. Tiny families of functions with random properties: A quality- 
size trade-off for hashing. Random Structures & v4^oni/imsril(4):315-343ri997. 

[GZ97] Oded Goldreich and David Zuckerman. Another proof that BPP C PH (and more). Elec- 
tronic Colloquium on Computational Complexity Technical Report TR97-045rSeptember 1997. 
http : //www . eccc . uni-tr ier . de/eccc. 



14 



[ILL89] Russell ImpagliazzorLeonid A. Levinrand Michael Luby. Pseudo-random generation from one- 
way functions (extended abstracts). In Proceedings of the Twenty First Annual ACM Symposium 
on Theory of ComputingTp&ges 12-24rSeattlerWashingtonri5-17 May 1989. 

[MR95] Rajeev Motwani and Prabhakar Raghavan. Randomized Algorithms. Cambridge University Pressr 
1995. 

[Nis96] Noam Nisan. Extracting randomness: How and why: A survey. In Proceedings, Eleventh Annual 
IEEE Conference on Computational Complexity^ pages 44-58rPhiladelphiarPennsylvaniar24- 
27 May 1996. IEEE Computer Society Press. 

[NT98] Noam Nisan and Amnon Ta-Shma. Extracting randomness: A survey and new constructions. 
Journal of Computer and System SciencesT1998. To appear in STOC '96 special issue. Preliminary 
versions in [Nis96] and [TS96]. 

[NW94] Noam Nisan and Avi Wigderson. Hardness vs randomness. Journal of Computer and System 
S«encesr49(2):149-167rOctober 1994. 

[NZ96] Noam Nisan and David Zuckerman. Randomness is linear in space. Journal of Computer and 
System S«encesr52(l):43-52rFebruary 1996. 

[Pip87] Nicholas Pippenger. Sorting and selecting in rounds. SIAM Journal on ComputingT16(6):1032- 
1038rDecember 1987. 

[RT97] Jaikumar Radhakrishnan and Amnon Ta-Shma. Tight bounds for depth-two superconcentrators. 
In 38th Annual Symposium on Foundations of Computer ScienceTp&ges 585-594rMiami Beachr 
Floridar20-22 October 1997. IEEE. 

[Sip88] Michael Sipser. Expandersrrandomnessror time versus space. Journal of Computer and System 
S«encesr36(3):379-383rjune 1988. 

[SSZ98] Michael SaksrAravind Srinivasanrand Shiyu Zhou. Explicit OR-dispersers with polylogarithmic 
degree. Journal of the 4CMT45(l):123-154rjanuary 1998. 

[SV86] Miklos Santha and Umesh V. Vazirani. Generating quasi-random sequences from semi-random 
sources. Journal of Computer and System S«encesr33(l):75-87rAugust 1986. 

[SZ98] Aravind Srinivasan and David Zuckerman. Computing with very weak random sources. To appear 
in SIAM Journal on ComputingT1998. Preliminary version in FOCS '94- 

[Tre98] Luca Trevisan. Simple and improved construction of extractors. Unpublished manuscriptrjuly 
1998. 

[TS96] Amnon Ta-Shma. On extracting randomness from weak random sources (extended abstract). In 
Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of ComputingTp&ges 
276-285rPhiladelphiarPennsylvaniar22-24 May 1996. 

[TS98] Amnon Ta-Shma. Almost optimal dispersers. In Proceedings of the 30th Annual ACM Symposium 
on Theory of ComputingTpages 196-202rDallasrTXrMay 1998. ACM. 

[Vaz84] Umesh V. Vazirani. Randomness, Adversaries, and Computation. PhD thesisr University of 
C aliforniarBerkeleyr 1984. 

[Vaz87a] Umesh V. Vazirani. Efficiency considerations in using semi-random sources (extended abstract). In 
Proceedings of the Nineteenth Annual ACM Symposium on Theory of ComputingTp&ges 160-168r 
New York Cityr25-27 May 1987. 

[Vaz87b] Umesh V. Vazirani. Strong communication complexity or generating quasirandom sequences from 
two communicating semirandom sources. Com6maioncar7(4):375-392ri987. 



15 



[VV85] Umesh V. Vazirani and Vijay V. Vazirani. Random polynomial time is equal to slightly-random 
polynomial time. In 26th Annual Symposium on Foundations of Computer ScienceTp&ges 417-428r 
PortlandrOregonr21-23 October 1985. IEEE. 

[WZ95] Avi Wigderson and David Zuckerman. Expanders that beat the eigenvalue bound: Explicit 
construction and applications. Technical Report CS-TR-95-2irUniversity of Texas Department 
of Computer SciencesIT995. To appear in Combinatorica. 

[Yao82] Andrew C. Yao. Theory and applications of trapdoor functions (extended abstract). In 23rd 
Annual Symposium on Foundations of Computer ScienceT pages 80-9irChicagorillinoisr3-5 
November 1982. IEEE. 

[Zuc96] David Zuckerman. Simulating BPP using a general weak random source. AlgorithmicdT 
16(4/5) :367-39irOctober/November 1996. 

[Zuc97] David Zuckerman. Randomness-optimal oblivious sampling. Random Structures & AlgorithmsT 
ll(4):345-367ri997. 

A Derandomizing the proof of Lemma 8 

In the analysis of the probabilistic choice of SjTwe showed that 



E 



y^l^n^-i 



j<i 



< p ■ (i - 1) 



By averagingfthis implies that there exists an ct\ € B\ such that 



E 



£ 2 I* 



c\Si\ 



j<i 



a\ = a.\ 



< P ■ (i ~ 1) 



(2) 



a\ = ai for every 



Sol 1 assuming we can efficiently calculate the conditional expectation E J2j<i 2\ SinSj \ 

ci\ € BiTwe can find the ci\ that makes Inequality 2 hold. Thenl 1 fixing such an ci\T another averaging 

argument implies that there exists an a 2 £ B 2 such that 



E 



^ 2 l Sin 



5,1 



j<i 



a\ = ai,ci2 = a.i 



< P ■ (i ~ 1) 



(3) 



Againr assuming that we can compute the appropriate conditional expectations! 1 we can find an a-i that 
makes Inequality 3 hold. Proceeding like thisfwe obtain cx\ , . . . , at such that 



E 



^ 2 l Sin 



5,1 



]<i 



a\ = a\ , CL2 = a>2, ■ ■ ■ ,ai = a>i 



< p ■ (i - 1) 



(4) 



But now there is no more randomness left in the experimentrand Inequality 4 simply says that ^2 iKi 2\ SinS ' I < 
p- (i — l)Tfor Si = {ai, . . . , a?}. To implement this algorithm for finding 5,rwe need to be able to calculate 
the conditional expectation 



E 



^ 2 l Sin 



^ I 



a\ = ai, . . . , a, = a 



]<i 
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for any i and a\, . . . , a,. If we let T = {a.\, . . . , a,}Tthen a calculation like the one in the proof of Lemma . 
for the unconditional expectation shows 



E 



j<i 



ai = ai,...,a.i = a. 



^ 2 l Tn ^'l (l 



]<i 



1 



\£/lnp] 



which can be easily computed. 
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